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Personnel  Performance  and 
Training  Technology 


FOREWORD 


Digital  computer  networks  currently  being  procured  and  fielded  by  the  U.S.  Army  have 
the  potential  to  significantly  improve  the  tactical  performance  of  combat  units.  Digitization  of 
the  force  promises  to  give  everyone  from  commander  to  frontline  Soldier  a  near  perfect  view  of 
the  battlefield,  thus  significantly  reducing  the  proverbial  “fog  of  war.”  Improved  information 
flow  makes  it  possible  to  quickly  plan,  coordinate,  and  execute  operations  at  an  increased 
operating  tempo,  so  that  tasks  that  once  took  hours  now  take  minutes. 

However,  the  promise  of  this  new  technology  relies  on  its  skilled  application.  Leaders 
and  Soldiers  must  learn  the  best  ways  to  apply  digitization  in  order  to  gain  the  anticipated  tactical 
advantages.  Formal  training  in  digital  systems  provides  an  essential  foundation  for  this  learning. 
Training  provides  digital  system  operators  and  leaders  with  the  basic  skills  needed  to  employ 
digitization. 

This  report  is  part  of  a  series  aimed  at  identifying  the  best  methods  for  measuring 
Soldiers’  proficiency  with  digital  technology.  It  measures  how  formal  training  affects  Soldier 
patterns  of  behavior  and  attitudes  towards  digitization.  The  results  of  this  investigation  can  be 
used  to  focus  scarce  training  resources  and  time  on  those  digital  skills  which  have  the  greatest 
impact  on  unit  performance. 

The  work  described  in  this  report  is  a  portion  of  research  task  234,  Defining  and 
Measuring  Digital  Skill  Proficiency,  sponsored  by  the  Program  Executive  Office  for  Simulation, 
Training,  and  Instrumentation  (PEOSTRI)  Project  Manager  for  Training  Devices.  The  results  of 
this  work  were  briefed  to  III  Corps’  Battle  Command  Training  Directorate  at  Fort  Hood,  Texas 
on  19  November,  2003. 


MICHAEL  G.  RUMSEY 


Acting  Technical  Director 
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HOW  FORMAL  TRAINING  AFFECTS  SOLDIER  ATTITUDES  AND  BEHAVIORS 
TOWARDS  DIGITIZATION 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army’s  use  of  networked  computer  systems  to  enhance  mission 
accomplishment,  collectively  known  as  digitization,  has  the  potential  to  effect  substantial 
improvements  in  how  military  operations  are  conducted.  However,  to  capitalize  on  the  benefits 
of  digitization,  units  must  invest  considerable  effort  to  train  Soldiers  in  the  skills  they  need  to 
exploit  the  advantages  of  digital  systems.  Given  that  training  time  is  often  a  scarce  resource,  it  is 
important  to  understand  how  training  affects  outcomes.  The  present  investigation  explores  how 
formal  classroom  training  affects  Soldiers’  behaviors  and  attitudes  towards  digitization. 

Procedure: 

A  set  of  questionnaires  was  administered  to  Soldiers  attending  formal  classroom  training 
for  the  Force  XXI  Battle  Command,  Brigade  and  Below  (FBCB2)  digital  system.  The 
participants  were  24  enlisted  Soldiers  and  12  officers  attending  either  FBCB2  Operator  New 
Equipment  Training  (OPNET)  or  leader  training.  One  questionnaire  was  administered  before 
training  began,  and  a  second  similar  questionnaire  after  training  was  complete.  The 
questionnaire  assessed  Soldiers’  opinions  of  the  usefulness  and  difficulties  associated  with  using 
FBCB2,  and  also  their  attitudes  associated  with  using  the  system.  Soldier  responses  before  and 
after  training  were  compared  to  assess  how  training  affected  their  attitudes  and  behaviors 
towards  digitization. 

Findings: 

Soldiers  generally  feel  FBCB2  is  useful,  and  is  worth  the  additional  effort  required  to 
learn  the  system.  They  reported  it  is  most  useful  for  planning,  preparation  for  combat,  and 
movement,  but  less  during  combat  (i.e.  enemy  contact).  These  phases  (planning,  preparation  for 
combat,  and  movement)  also  were  reported  to  entail  the  highest  workload.  Soldiers  ranked  the 
most  useful  planning  capabilities  as  sending  and  receiving  messages  and  overlays,  and  route 
planning.  The  non-planning  related  features  ranked  as  most  useful  were  sending  and  receiving 
messages,  navigating  to  a  location,  checking  the  location  of  enemy  units,  and  avoiding  threats 
such  as  minefields.  Overall,  Soldiers  reported  that  those  features  of  FBCB2  which  involved  the 
highest  workload  and  were  most  difficult  to  work  with  were  also  those  rated  the  most  useful.  In 
addition.  Soldiers’  responses  indicated  they  are  unsure  of  the  accuracy  of  FBCB2  elements, 
possibly  because  this  topic  is  not  covered  in  training. 

Soldiers’  responses  after  training  were  significantly  different  than  before  training  for  the 
majority  of  the  topics  covered  in  the  class.  However,  topics  not  covered  during  the  course 
showed  little  change  from  pre-  to  post  training.  Agreement  among  Soldiers  about  aspects  of 
digitization  was  generally  higher  after  training  than  before,  although  for  some  questions 
agreement  was  fairly  low  both  before  and  after  training. 
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The  results  also  seem  to  indicate  that  training  may  help  Soldiers  avoid  maladaptive 
behavior  patterns  which  have  been  identified  in  other  areas.  Behaviors  such  as  disuse  or  over¬ 
reliance  on  automated  systems,  a  significant  problem  in  other  areas,  were  relatively  rare  in 
Soldiers  attending  formal  training. 

Utilization  of  Findings: 

These  findings  can  be  used  to  show  commanders  the  value  of  formal  training  over  other 
training  methods,  such  as  on-the-job  training  or  discovery  learning.  They  show'  that  formal 
training  provides  more  benefits  for  Soldiers  than  simply  learning  to  operate  digital  systems. 
Formal  training  affects  Soldiers’  willingness  to  employ  digitization  and  may  encourage  them  to 
use  digital  systems  more. 

The  findings  can  also  be  used  by  trainers  to  decide  what  elements  of  digitization  and 
digital  systems  are  most  difficult  to  learn  and  require  the  most  training  emphasis.  In  addition, 
since  some  of  the  questions  asked  Soldiers  about  the  usability  and  workload  associated  with 
different  digital  tasks  and  tools,  this  information  can  be  used  by  program  managers  to  improve 
system  usability. 
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HOW  FORMAL  TRAINING  AFFECTS  SOLDIER  BEHAVIORS  AND  ATTITUDES 
TOWARDS  DIGITIZATION 


Introduction 

This  research  investigates  how  Soldiers’  attitudes  and  behaviors  towards  digital  systems 
change  as  they  undergo  formal  training.  The  effort  had  several  objectives.  The  first  was  to 
determine  how  formal  training  affected  Soldiers  attitudes  and  behaviors  towards  digital  systems, 
and  also  how  it  changed  their  use  of  the  systems.  The  second  was  to  identify  what  capabilities  of 
digital  systems,  in  this  case  the  Force  XXI  Battle  Command,  Brigade  and  Below  (FBCB2) 
Soldiers  found  more  or  less  useful.  The  last  objective  was  to  identify  if  Soldiers  exhibited  certain 
maladaptive  and  adaptive  behavior  patterns  towards  digital  systems  which  have  been  identified 
in  other  areas  where  users  interact  with  automated  systems.  The  effort  did  not  assess  training 
methods,  rather,  it  measured  the  results  of  the  training. 

Before  describing  how  this  research  was  conducted,  I  will  provide  some  background  on 
U.S.  Army  digitization,  how  digitization  is  trained,  and  also  discuss  maladaptive  and  adaptive 
patterns  of  behavior  towards  automation. 

Background 

Digitization 

In  the  U.S.  Army,  digitization  consists  of  networked  computer  systems  which  are  used  to 
improve  communication,  mission  planning,  and  integration  among  Battlefield  Operating  Systems 
(BOS).  There  are  a  number  of  digital  systems,  most  of  which  are  tailored  to  support  a  specific 
BOS,  but  which  can  share  information  with  other  systems  (for  a  more  detailed  description  of 
digitization,  see  Barnett,  Meliza  &  McCluskey,  2001). 

The  digital  system  studied  in  this  research  was  the  FBCB2  system.  FBCB2  is  used  by 
lower  echelon  (as  the  name  implies,  below  brigade  level)  maneuver  units  both  in  Tactical 
Operations  Centers  (TOC)  and  in  individual  combat  vehicles.  In  TOCs,  FBCB2  is  used  to  plan 
missions,  send  orders  and  battlefield  graphics,  and  coordinate  combat  support  functions.  On 
vehicles,  it  is  used  to  receive  orders  and  graphics,  plan  routes  and  navigate  to  specific  locations, 
as  well  as  enhancing  Soldier’s  situation  awareness  of  the  battle  space.  Most  vehicles  with 
FBCB2  are  able  to  automatically  report  their  position  using  onboard  Global  Positioning  System 
(GPS)  equipment.  Thus,  commanders  and  Soldiers  can  view  the  position  of  their  vehicles  and 
others  in  their  unit  in  real  time.  An  example  of  the  FBCB2  display  is  shown  in  Figure  1. 

Digitization  Training 

Formal  training  for  digital  systems  is  conducted  at  the  Battle  Command  Training  Center 
(BCTC)  at  Fort  Hood,  TX.  BCTC  provides  Operator  New  Equipment  Training  (OPNET),  leader 
training,  and  staff  officer  training  for  a  number  of  different  digital  systems.  OPNET  for  FBCB2 
is  normally  approximately  40  hours  of  training  conducted  over  five  days;  36  hours  of  instruction 
with  a  four-hour  practical  examination.  There  are  usually  three  instructors  per  class;  one 
instructor  presents  the  material  while  the  other  two  assist  students.  The  leader  course  is  similar 
to  OPNET  training,  but  includes  more  tactical  considerations. 

The  class  normally  consists  of  20-25  Soldiers  using  desktop  computers  (training  systems 
or  “white  boxes”)  with  FBCB2  software.  The  computers  are  connected  over  a  local  area  network 
(LAN)  so  that  each  classroom  simulates  an  interconnected  unit.  For  one  8-hour  block  of 
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instruction  the  students  moved  to  another  classroom  to  work  with  real  FBCB2  hardware  (“green 
boxes”)  to  leam  equipment  inspection  and  pre-combat  checks.  The  desktop  computers  use  a 
mouse  and  keyboard  for  input,  while  the  actual  systems  use  a  touch  screen  and  track  ball-like 
device.  Although  “white  boxes”  have  different  input  methods  than  “green  boxes,”  Soldiers 
report  no  problem  switching  between  input  devices. 
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Figure  1.  Example  of  Force  XXI  Battle  Command  Brigade  and  Below  (FBCB2)  display. 

Most  of  the  blocks  of  instruction  are  followed  by  a  practical  exercise  where  the  students 
are  able  to  practice  what  they  learned.  The  blocks  of  instruction  and  exercises  follow  a  natural 
progression  including  system  startup,  pre-combat  checks,  movement,  assembly  area  operations, 
and  actions  on  the  objective.  The  training  assumes  the  Soldiers  have  a  basic  tactical  knowledge 
including  familiarity  with  reports,  map  symbology,  and  battlefield  graphics.  It  also  assumes  a 
basic  knowledge  of  computer  operation,  such  as  using  a  computer  mouse  and  keyboard,  as  well 
as  fanailiarity  with  common  desktop  computer  terminology.  Previous  experience  with  common 
desktop  office  software  is  helpful  since  the  interface  uses  many  conventions  from  office 
software,  particularly  the  drawing  tools  used  to  draw  battlefield  graphics. 

However,  many  Soldiers  work  with  FBCB2  before  they  receive  formal  training  because 
the  formal  training  is  often  difficult  to  squeeze  into  the  unit  training  schedule.  They  often  leam 
about  FBCB2  through  on-the-job-training  (OJT)  or  through  a  tutorial  included  with  some  of  the 
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FBCB2  software.  Thus,  some  of  the  Soldiers  attending  BCTC  classes  have  digitization 
experience  prior  to  classroom  training. 

Patterns  of  Behavior  Towards  Automation 

Research  has  shown  that  people’s  attitudes  and  behaviors  towards  automation  tend  to 
form  patterns.  A  decade  of  study  in  commercial  aviation  shows  certain  patterns  in  how  people 
react  to  the  introduction  of  automation  into  their  work  structure,  and  how  they  interact  with  the 
automated  systems.  Many  of  these  behavior  patterns  interfere  with  the  proper  use  of  automation. 
Some  users  may  avoid  using  automation  all  together,  and  thus  lose  the  advantages  automation 
offers.  Conversely,  other  users  may  rely  too  much  on  automation,  which  often  leads  them  into 
unpleasant  or  dangerous  situations  (Parasuraman  &  Riley,  1997).  Similar  patterns  are  beginning 
to  show  in  other  areas  where  automation  has  been  introduced,  such  as  surface  transportation 
(Parasuraman  &  Riley,  1997)  and  medicine  (Sheridan  &  Thompson,  1994). 

Not  surprisingly,  initial  studies  and  field  observation  seem  to  indicate  Soldier’s  attitudes 
and  behaviors  towards  digitization  follow  similar  patterns.  A  recent  study  by  Barnett  &  Meliza, 
(2003)  showed  concerns  voiced  by  Soldiers  about  digitization,  such  as  erosion  of  skills,  system 
reliability,  and  increased  workload,  mirror  concerns  expressed  by  commercial  air  crews 
regarding  automation. 

If  Soldier’s  behaviors  towards  digitization  follow  known  patterns  of  behavior  in  relation 
to  automation,  this  information  can  be  useful  in  terms  of  integrating  digitization  into  U.S.  Army 
units.  Commanders,  trainers,  and  evaluators  can  use  this  knowledge  to  help  them  assess  their 
unit’s  level  of  digital  training  and  tailor  training  to  be  most  effective. 

Typical  behavior  patterns.  A  review  of  previous  research  and  theory  suggests  that  users 
seem  to  exhibit  certain  behavior  patterns  in  response  to  working  with  automated  systems  (see 
Parasuraman  &  Riley,  1997).  These  patterns  may  reflect  their  attitudes  towards  automation,  and 
may  change  as  attitudes  change.  For  example,  people  are  often  skeptical  of  automated  systems 
at  first,  and  will  resist  learning  about  them  and  avoid  using  them  if  they  can.  Others  will  use  the 
systems,  but  only  in  a  limited  way.  Sometimes  people  with  more  computer  experience  may 
actually  rely  too  much  on  automation,  which  often  causes  its  own  set  of  problems.  Conversely, 
other  users  rely  on  it  in  a  more  realistic  way,  possibly  due  to  more  experience  or  expertise  with 
the  systems. 

Thus,  the  four  patterns  of  behavior  of  interest  in  this  research  are: 

•  Avoidance  or  disuse  behaviors 

•  Limited  use  behaviors 

•  Ovemse  or  over-reliance  behaviors 

•  Expert  behaviors 

In  addition  to  the  other  research  questions,  the  present  research  sought  to  identify  if 
Soldiers  exhibited  any  of  these  four  behavior  patterns.  Previous  field  observations  had  identified 
disuse  and  limited  use  behavior  patterns  in  some  digital  units.  I  wanted  to  see  if  Soldiers 
attending  formal  digitization  training  exhibited  any  of  these  behaviors,  and,  if  so,  how  the  formal 
training  affected  the  behaviors. 
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Method 


Participants 

Participants  were  36  Soldiers  attending  classes  at  the  BCTC.  The  sample  contained  24 
enlisted  Soldiers  from  Sergeant  (E5)  to  Master  Sergeant  (E8),  and  12  officers  from  Second 
Lieutenant  (01)  to  Major  (04).  Soldiers  below  E5  were  not  selected  to  participate  because  it 
was  felt  their  participation  would  take  up  too  much  class  time.  There  was  a  wide  range  of 
Military  Occupational  Specialty  (MOS)  codes,  indicating  participants  came  from  a  diverse 
background.  Tables  1  and  2  presents  the  list  of  MOS’s. 

Table  1 .  List  Of  Enlisted  Participants  ’  MOS 


Enlisted  MOS 

MOS  Series 

Functional  Area 

MOS  Series 

Functional  Area 

11B/M 

Infantry 

75H 

Administration 

12B 

Engineering 

77F 

Petroleum 

14S 

Air  Defense 

79S 

Recruiting 

19K 

Armor 

88M 

transportation 

44B  52C  63B/H/Z 

Maintenance 

95B 

Military  Police 

Table  2.  List  Of  Officer  Participants  ’  MOS 


Officer  MOS 

MOS  Series 

Branch 

MOS  Series 

Branch 

12A/C 

Engineer 

74A 

Civil  Affairs 

14B 

Air  Defense 

91A 

Chemical  Corps 

25A 

Signal  Corps 

92A/D 

Ordinance 

38A 

Quartermaster 

For  enlisted  Soldiers,  the  range  of  years  of  service  ranged  from  2  years  of  service  to  over 
20  years  (M=\  1 .6,  SD=5.S6).  Officers'  years  of  service  ranged  from  less  than  a  year  to  22  years 
(Af=7.77,5D=7.18). 

Participant  Computer  Experience.  When  asked  about  experience  with  FBCB2  and 
general  computer  experience,  enlisted  Soldiers  reported  they  had  used  FBCB2  for  less  than  12 
months  {M=\ .46, 5£)=2.87)  with  many  (15)  reporting  they  have  no  experience  with  FBCB2. 
When  asked  how  many  hours  a  month  they  use  FBCB2,  most  (16)  reported  they  did  not  use  it, 
while  those  that  did  averaged  2.6  hours  a  month  (SD=5.34).  Officers  reported  they  used  FBCB2 
six  months  or  less  (Af=l .  1 ,  SD=\ .78),  with  the  majority  (7)  reporting  they  had  not  used  it  yet. 

The  distributions  for  all  of  these  statistics  were  positively  skewed,  probably  because  of  the  large 
number  who  reported  no  FBCB2  use. 

Participants  were  also  asked  how  many  hours  a  week  they  used  computers  at  work  and  at 
home,  and  also  how  many  hours  they  played  computer  games.  Enlisted  Soldiers  reported  they 
used  computers  at  work  an  average  of  1 5.3  hours  a  week  {SD=\ 8.8),  and  at  home  an  average  of 
12.4  hours  (5'Z)=13.4).  They  also  reported  they  played  computer  games  an  average  of  3.7  hours  a 
week  {SD=1.61).  Officers  said  they  used  computers  at  work  an  average  of  22.5  hours  a  week 
{SD~\6.1)  and  12.8  hours  at  home  {SD=9.7).  Officers  also  said  they  played  computer  games  an 
average  of  4.5  hours  a  week  (5D=5.7).  Again,  these  statistics  were  positively  skewed,  which  can 
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probably  be  attributed  to  the  number  of  participants  who  reported  no  computer  use  at  work  or  at 
home.  Table  3  provides  a  comparison  of  how  many  hours  a  week  Soldiers  who  responded  to 
these  questions  use  PCs  at  work,  at  home,  and  for  gaming. 


Table  3.  Hours  per  Week  Respondents  Use  Personal  Computers  at  Work,  at  Home,  and  Gaming 


Respondent 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

1  14 

15 

16 

17 

1  18  1 

Hours/Work 

60 

ESI 

40 

30 

25 

25 

20 

15 

10 

10 

10 

8 

0 

0 

0 

m 

Hours/Home 

4 

m 

m 

5 

30 

20 

10 

20 

20 

40 

15 

4 

6 

1 

7 

1 

1 

Dl 

1 

3 

I  15  1 

2 

15 

0 

10 

0 

3 

30 

4 

0 

14 

0 

1 

1 

m 

A  cross  correlation  matrix  was  constructed  to  identify  relationships  between  participant 
demographics  and  both  digital  and  general  computer  experience.  Scatterplots  showed  the 
variables  were  not  linearly  related,  thus  Spearmans’  Rho  was  chosen  over  the  Pearson  Product 
Moment  Correlation.  Significant  correlations  included  a  positive  correlation  between  rank  and 
hours  of  computer  use  at  work  (rs  =  .381,  p.<  .026),  a  negative  correlation  between  time  in 
service  and  playing  computer  games  (rs  =  -.447,  p.<  .008),  a  high  positive  correlation  between 
months  of  using  FBCB2  and  hours  a  week  using  FBCB2  (rs  =  .935,  p.<  .000),  and  a  positive 
correlation  between  home  computer  use  and  playing  computer  games  (rs  =  .676,  p.<  .000). 

Materials 

The  materials  consisted  of  a  set  of  questionnaires,  one  administered  pre-  and  one  post¬ 
training.  In  addition  to  demographics,  the  questions  were  organized  into  four  sections.  The  first 
section  asked  Soldiers  questions  about  the  usability,  workload,  and  accuracy  of  FBCB2  and  had 
them  rank  order  a  set  of  responses.  The  second  section  consisted  of  Likert-type  questions  that 
asked  Soldiers  to  agree  or  disagree  with  statements  related  to  typical  behaviors  towards 
automation.  Section  three  asked  questions  about  how  Soldiers  monitor  digital  systems  and  had 
Soldiers  select  from  a  series  of  responses.  The  last  section  asked  several  free  response  questions 
about  system  monitoring  and  about  which  tasks  were  the  most  difficult  to  perform. 

The  questionnaires  also  solicited  demographic  information  as  well  as  information  on  the 
individual’s  digital  and  general  computer  experience.  Similar  questions  were  asked  before  and 
after  training.  The  order  of  questions  on  the  post-training  questionnaire  was  changed,  and  some 
of  the  questions  were  reworded  to  reduce  the  chance  of  the  respondents  being  influenced  by 
remembering  the  question  from  the  pre-training  questionnaire.  The  questions  are  discussed  in 
the  results  section.  Examples  of  the  questionnaires  are  in  Appendix  B. 

Procedure 

The  sets  of  questionnaires  were  distributed  among  the  participants  at  the  beginning  of  the 
classroom  training.  We  decided  that  it  would  be  too  time  consuming  and  disruptive  of  the  class 
to  collect  the  first  questionnaire  and  then  distribute  the  second  questionnaire  at  the  end  of 
training.  This  would  require  the  participants  to  remember  participant  numbers  which  would  put 
an  undue  burden  on  both  the  participants  and  instructors.  Therefore,  we  delivered  both 
questionnaires  in  an  envelope  which  was  kept  at  the  student  workstation  throughout  the  training 
To  ensure  the  questionnaires  were  completed  in  the  correct  order,  they  were  clearly  marked  on 
the  first  page  with  instructions  about  the  order  of  completion.  In  addition,  the  second 
questionnaire  was  printed  on  paper  of  a  different  color  (yellow)  than  the  first  questionnaire 
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(white).  The  second  questionnaire  was  also  folded  and  sealed  to  further  reduce  the  probability  of 
Soldiers  competing  the  questionnaires  in  the  wrong  order. 

At  the  beginning  of  the  class,  participants  were  briefed  on  the  purpose  of  the  research  and 
reminded  that  their  participation  was  voluntary.  Participants  were  then  instructed  to  complete 
the  first  (white)  questionnaire  and  return  it  to  the  envelope.  After  training  was  completed  at  the 
end  of  the  week,  they  were  instructed  to  remove  the  second  (yellow)  questionnaire,  break  the 
seal,  complete  the  questionnaire,  and  return  it  to  the  envelope.  The  envelopes  were  then 
collected. 

Both  questionnaires  and  the  envelope  were  identified  by  a  participant  number. 
Participants  were  briefed  that  they  could  mark  their  names  on  the  envelope  if  they  wished  to 
prevent  mix-ups  but  not  to  put  their  names  on  the  questionnaires.  Once  the  envelopes  were 
returned,  we  separated  the  questionnaires  from  the  envelopes  to  ensure  participant  anonymity. 
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Results 


The  questionnaire  was  in  four  sections,  each  soliciting  information  in  different  ways. 
Each  of  these  sections  will  be  discussed  in  turn. 

Questionnaire  Section  One:  Usefulness  ofFBCB2 

The  first  section  asked  Soldiers  questions  about  FBCB2  and  had  them  rank  order  a  set  of 
responses.  The  questions  sought  to  identify  how  Soldiers  ranked  the  importeince  of  aspects  of 
FBCB2,  and  also  how  their  view  of  the  importance  of  these  functions  changed  with  training. 
Figure  2  shows  an  example  of  this  type  of  question.  The  complete  questionnaires  are  found  in 
Appendix  B. 


1 .  What  do  you  consider  the  most  useful  planninq  capabilities  with  FBCB2?  Please  rank  order  from  most 

useful  (number  11  to  least  useful 

Plan  a  route 

Plan  support  measures 

Send/receive  messaaes/olans  /overlays 

Line-of-sight  tools 

Other 

Figure  2.  Example  of  question  from  section  one  of  the  questionnaire. 

There  are  three  questions  I  want  to  answer  about  this  section;  (1)  how  did  Soldiers  rank 
the  statements,  (2)  how  much  agreement  was  there  among  the  Soldier’s  ratings,  and  (3)  was  there 
a  significant  difference  between  how  a  statement  was  rated  pre-test  and  post-test? 

To  answer  the  first  question,  I  initially  looked  at  the  mode  of  the  rankings  of  statements. 
Unfortunately,  many  of  the  statements  were  multi-modal  with  two  and  sometimes  three 
statements  receiving  the  same  frequency  of  rankings.  Therefore,  I  calculated  the  mean  rank 
using  Kendall’s  W  test.  Kendall’s  W,  or  Kendall’s  Coefficient  of  Concordance,  determines  the 
agreement  among  raters  and,  as  a  byproduct,  calculates  the  mean  rank  of  each  statement.  Mean 
rank  is  more  descriptive  than  mode  in  this  case  because  it  avoids  the  problem  of  multiple  modes 
and  provides  a  better  idea  of  how  statements  were  rated  by  the  population.  Since  the  participants 
ranked  the  statements  from  1  (highest  rank)  to  k  (lowest  rank),  statements  with  lower  numbers 
were  rated  higher. 

In  addition,  Kendall’s  W  also  answered  our  second  question  about  how  much  agreement 
there  was  among  the  raters.  Kendall’s  W  statistic  rates  agreement  as  values  from  zero  to  one, 
with  zero  being  no  agreement  among  raters,  to  one  being  perfect  agreement.  The  test  for 
significance  of  the  W  statistic  is  a  chi-square  which  determines  how  the  calculated  statistic 
differs  from  chance.  (Siegel,  1956).  The  results  of  these  analyses  are  discussed  next. 

Question  1.  The  first  question  asked,  “What  do  you  consider  the  most  useful  planning 
capabilities  with  FBCB2?”  The  response  statements  were,  “Send/receive  messages/plans/ 
overlays,”  “Plan  a  route,”  Line-of-sight  tools,”  and  “Plan  support  measures.”  The  analysis  of 
this  question  is  summarized  in  Table  4,  which  shows  the  mode,  mean  rank,  and  Kendall’s  W  for 
each  statement  both  before  and  after  training.  The  Soldiers  generally  ranked  the  usefulness  of 
these  capabilities  as  shown  in  the  Table,  except  that  “Plan  support  measures”  got  a  higher  rating 
than  “Line-of-sight  tools”  on  the  pre-test,  but  was  rated  lower  on  the  post-test.  Kendall’s  W 
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showed  there  was  moderately  low  agreement  about  the  rankings  on  the  pre-test,  but  agreement 
was  much  higher  for  the  post-test.  In  both  cases,  the  statistic  differed  significantly  from  chance 
and  exceeded  the  p<.05  confidence  level. 


Table  4.  How  Soldiers  Ranked  The  Usefulness  OfFBCBl  Planning  Capabilities  (Question  1) 


Statement  “ 

Pre-training 

Post-training 

Mode 

M  Rank^ 

Mode 

M  Rank 

W 

Send/receive  messages/overlays 

1 

1.70 

.247 

(X'=20.0, 

df=3, 

p<.000) 

1 

1.47 

.398 

(X=35.8, 

df=3, 

p<.000) 

Plan  a  route 

1 

2.26 

2 

2.23 

Line-of-sight  tools 

4 

3.07 

4 

3.10 

Plan  support  measures 

3 

2.96 

1 

3.20 

^Sialemenls  are  listed  in  post-training  rank  order. 
^Mean  Rank  from  Kendall’s  W  test. 

Kendall’s  Coefficient  of  Concordance  (VV). 


Question  2.  The  second  question  was,  “What  do  you  consider  the  most  useful  FBCB2 
capabilities  other  than  planning?”  and  asked  respondents  to  rank  order  statements  from  most 
useful  (number  one)  to  least  useful.  The  statements  are  listed  in  Table  5,  along  with  the  analysis 
of  the  data.  The  Kendall’s  W  shows  there  was  little  agreement  among  respondents  either  before 
or  after  training,  although  there  was  more  agreement  after  training  than  before.  The  modes  show 
the  responses  to  be  multi-modal,  and  there  is  little  variance  between  the  mean  ranks,  both  of 
which  probably  reflect  the  lack  of  agreement  among  Soldiers  as  to  the  usefulness  of  FBCB2 
capabilities. 


Table  5.  How  Soldiers  Ranked  The  Usefulness  OfFBCBl  Capabilities  Other  Than  Planning 
( Question  2 ) 


Statement  “ 

Pre-training 

Post-training 

Mode 

M  Rank” 

Vi^c 

Mode 

Af  Rank 

W 

Send/receive  messages 

3 

4.11 

.126 

(X=20.0, 

df=6, 

p<.002) 

3 

3.15 

.203 

(X=36.5, 

df=6, 

p<.000) 

Navigate  to  a  location 

4 

3.83 

1 

3.27 

Check  location  of  enemy  units 

2 

3.52 

1 

3.47 

Avoid  threats  (minefields,  etc.) 

1 

3.17 

2 

3.60 

Check  location  of  subordinate  unit 

4 

3.37 

5 

3.82 

Logistics/personnel  reporting 

7 

5.28 

6 

4.97 

Check  status  of  subordinate  unit 

a  _ _  1-  .  1-  .. 

7 

4.72 

7 

5.73 

“  Statements  are  listed  in  post-training  rank  order. 
’’Mean  Rank  from  Kendall’s  W  test. 

'  Kendall’s  Coefficient  of  Concordance  (W). 


One  possibility  for  the  lack  of  agreement  is  that  the  participants  came  from  a  variety  of 
different  backgrounds  and  therefore  used  FBCB2  for  different  purposes.  There  were  17  different 
MOS’s  among  36  participants,  which  is  a  clearly  diverse  group.  This  may  account  for  them 
rating  the  usefulness  of  certain  functions  differently.  To  test  this  idea,  I  ran  a  Kendall’s  Won  a 
select  group  of  Soldiers  with  similar  MOS’s.  I  chose  participants  with  infantry  (11  series)  and 
armor  (19  series)  MOS’s  since  these  Soldiers  should  use  FBCB2  for  similar  purposes. 
Unfortunately,  even  by  combining  infantry  and  armor  MOS’s,  the  n  was  still  too  small  to  give  a 
confidence  interval  better  than  p<.05.  Thus  this  test  gave  us  no  useful  information. 
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Question  3.  This  question  asked  Soldiers  “When  is  roCB2  most  useful  to  you?”  and 
provided  a  list  of  combat  phases  for  them  to  rank  order.  The  phases  and  analysis  of  the  data  are 
shown  in  Table  6.  The  table  shows  there  was  moderate  agreement  prior  to  training,  but 
agreement  was  considerably  higher  post-training.  Post-test  modes  and  mean  ranks  show  that 
Soldiers  feel  that  FBCB2  is  most  useful  during  planning,  preparation  for  combat,  and  movement 
phases,  whereas  it  is  least  useful  during  assembly  area  operations  and  actions  on  the  objective. 

Table  6.  How  Soldiers  Ranked  When  FBCB2  Was  Most  Useful  ( Question  3) 


Statement  “ 

Pre-training 

Post-training 

Mode 

Mode 

M  Rank 

W 

Planning 

1 

.230 

(X=38.5, 

df=6, 

p<.000) 

1 

BQQffI 

.402 

(X=72.3, 

df=6, 

p<.000) 

Preparation  for  combat 

4 

3.93 

2 

IBESH 

Movement 

1 

2.29 

3 

2.93 

Attack 

4 

4.07 

6 

4.68 

Defense 

6 

4.57 

4 

4.90 

Assembly  area  operations 

7 

7 

Actions  on  the  objective 

7 

5.04 

7 

5.35 

“Statements  are  listed  in  post-training  rank  order. 
’’ Mean  Rank  from  Kendall’s  IV  test. 

‘  Kendall’s  Coefficient  of  Concordance  (W). 


Question  4.  This  question  asked  Soldiers,  “When  is  roCB2  workload  the  highest?”  and 
provided  the  same  list  of  combat  phases  as  the  previous  question.  The  data  analysis  is 
summarized  in  Table  7.  The  analysis  shows  that  rater  agreement  was  fairly  low  both  before  and 
after  training,  although  it  did  increase  slightly  after  training.  It  is  interesting  to  note  that  the 
rankings  of  combat  phases  for  this  question  are  roughly  similar  to  the  previous  question  which 
asked  about  when  FBCB2  was  most  useful.  Although  the  ranking  for  the  first  two  phases  in 
question  3  are  reversed  for  question  4,  the  mode  and  mean  rank  statistics  show  these  were  very 
close  in  ranking.  The  similarity  between  the  ranking  of  these  phases  suggests  that  when  FBCB2 
is  most  useful,  it  also  has  the  highest  workload. 


Table  7.  How  Soldiers  Ranked  When  FBCB2  Workload  Was  The  Highest  (Question  4) 


Statement  “ 

Pre-training 

Post-training 

Mode 

Mode 

MRank 

w 

Preparation  for  combat 

3 

3.77 

.213 

(X=33.1, 

df=6, 

p<.000) 

2 

2.42 

.260 

(x'=40.5, 

df=6, 

p<.000) 

Planning 

1 

2.81 

1 

2.62 

Movement 

1 

2.62 

3 

3.96 

Attack 

6 

4 

4.23 

Defense 

7 

4.96 

5 

4.50 

Assembly  area  operations 

7 

4.92 

7 

BIBI 

Actions  on  the  objective 

7 

4.88 

7 

wmxam 

“  Statements  are  listed  in  post-training  rank  order. 
** Mean  Rank  from  Kendall’s  tV test. 

'  Kendall’s  Coefficient  of  Concordance  (W). 
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Question  5.  The  last  question  in  this  section  asked,  “On  the  FBCB2  display,  which  of  the 
following  is  usually  closest  to  ground  truth  (most  accurate)?”  The  responses  and  the  analysis  of 
the  data  is  shown  in  Table  8.  For  this  question  the  W  statistic  for  pre-training  agreement  docs  not 
meet  the  p<.05  confidence  level.  However,  the  post-training  W  statistic  does  provide  .sufficient 
confidence,  and  shows  that  even  after  training  the  agreement  among  Soldiers  was  low.  This 
analysis  suggests  Soldiers  are  unsure  about  the  accuracy  of  FBCB2. 


Table  8.  How  Soldiers  Ranked  The  Accuracy  Of  FBCB2  Display  Elements  (Question  5) 


Statement  “ 

Pre-training 

Post-training 

Mode 

M  Rank^ 

Mode 

M  Rank 

W 

Friendly  icons  (center  of  mass) 

3 

3.19 

.047 

(X=4.9, 

df=5, 

p<.423) 

1 

2.57 

.161 

(X=16.9, 

df=5, 

p<.005) 

Terrain  features 

1 

3.48 

1 

3.05 

Friendly  icons  (individual 
platforms) 

1 

2.95 

2 

3.33 

Overlays 

3 

3.86 

1 

3.43 

Line-of-sight  analysis 

3 

3.48 

2 

3.86 

Enemy  icons 

a  _ _  1*  ,  i  *  .  .  •  .  . 

5 

4.05 

6 

4.76 

*  Statements  are  listed  in  post-training  rank  order. 
‘’Mean  Rank  from  Kendall’s  W test. 

Kendall’s  Coefficient  of  Concordance  (W). 


Differences  after  training.  I  also  wanted  to  determine  if  Soldier  ratings  of  usefulness, 
workload,  and  accuracy  of  FBCB2  changed  after  training.  Therefore,  for  every  question  I 
compared  how  Soldiers  ranked  each  item  before  training  with  the  ranking  they  gave  it  after 
training  using  the  Wilcoxon  Signed  Ranks  test.  The  Wilcoxon  Signed  Ranks  test  is  a 
nonparametric  test  which  determines  if  a  significant  difference  exists  between  matched  pairs  of 
scores  using  Z  scores  (Siegel,  1956).  For  all  questions  in  this  section  I  were  interested  in 
differences  which  exceeded  the  p<.05  confidence  level  (two-tailed). 

Question  one  had  no  differences  which  exceeded  the  set  confidence  level.  Question  two 
asked  which  were  the  most  useful  FBCB2  capabilities  other  than  planning.  There  were  two 
responses  which  differed  significantly;  “Check  status  of  subordinate  units”  (Z=-2.45,  p<.014), 

and  “Send/receive  messages”  (Z=  -2.21,  p<.027),  both  of  which  were  ranked  as  less  important 
after  training. 

Question  three,  which  asked  when  FBCB2  was  most  useful  also  had  two  responses  which 
were  ranked  differently  after  training.  “Planning”  (Z=  -2.58,  p<.01)  and  “preparation  for 
combat”  (Z=-3.24,  p<.001)  were  both  rated  higher  after  training.  Question  four  asked  when 
FBCB2  workload  was  the  highest.  “Preparation  for  combat”  (Z=-3.12,  p<.002)  increased  in 
ranking  after  training,  and  “movement”  (Z=-3.34,  p<.001)  decreased  after  training.  Question 
five  only  had  one  statement  which  differed  significantly.  This  question  asked  about  the  accuracy 
of  display  elements  in  FBCB2.  Soldiers  ranked  “friendly  icons  (center  of  mass)”  more  highly 
after  training  than  before  (Z=-2.56,  p<.01 1). 

Questionnaire  Section  Two:  Attitudes  and  Behaviors  Towards  FBCB2 

The  second  section  consisted  of  Likert-type  questions  that  asked  Soldiers  to  agree  or 
disagree  with  statements.  Soldiers  were  given  nine  statements  and  asked  whether  they  strongly 
disagreed,  disagreed,  agreed,  or  strongly  agreed  with  the  statement.  A  fifth  option  was  “Don’t’ 
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know.”  The  nine  questions  were  cross-correlated  using  Spearman’s  Rho  to  ensure  there  were  no 
problems  of  singularity,  that  is,  questions  which  measure  the  same  construct  (Tabachnick  & 
Fidell,  1996).  The  highest  correlation  (rs  =  .676,  p.<.000)  was  between  questions  7  (“FBCB2 
really  helps  me  do  my  job.”)  and  12  (“Even  when  it’s  not  working  perfectly,  digitization  is  better 
than  analog.”)  on  the  post-training  questionnaire.  Since  none  of  the  questions  were  highly 
correlated  (.9  or  greater)  with  each  other,  I  assumed  there  was  no  singularity. 

This  section  sought  to  identify  Soldier  attitudes  and  behaviors  towards  digitization  and 
determine  if  they  are  similar  to  attitudes  and  behaviors  reported  towards  automated  systems  in 
other  areas.  The  behaviors  of  interest  were  avoidance  or  disuse  behaviors,  limited  use  behaviors, 
and  overuse  behaviors.  For  a  discussion  of  use,  disuse,  and  overuse  of  automation,  see 
Parasuraman  and  Riley  (1997).  In  addition,  several  of  the  questions  sought  to  identify  Soldiers 
who  exhibited  behaviors  thought  to  show  a  deeper  understanding  of  automation  strengths  and 
weaknesses  (expert  behaviors). 

The  first  statement  in  this  section  was  “Sometimes  the  FBCB2  does  things  for  no  reason 
or  that  I  don’t  understand.”  This  question  seeks  to  measure  disuse/avoidance  behaviors,  and  it 
was  expected  that  Soldiers  who  agree  with  this  statement  would  tend  to  avoid  using  FBCB2. 
Figure  3  shows  a  bar  graph  comparing  Soldier  responses  pre-  and  post-training.  The  dark  bars 


"Sometimes  the  FBCB2  does  things  for  no  reason  or  that  I  don't  understand" 


I  gPre-training  □Post-training 


are  pre-training  responses  and  the  light  bars  show  post-training  responses. 

Figure  3.  Comparison  of  frequencies  of  Soldier  responses  before  and  after  training  for  statement 
6,  “Sometimes  the  FBCB2  does  things  for  no  reason  or  that  I  don’t  understand.” 


The  figure  shows  that  before  training  responses  were  fairly  evenly  split  between 
agreement  (56%)  and  disagreement  (44%),  and  also  that  20  out  of  36  respondents  answered 
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disagreeing  with  the  statement.  This  suggests  Soldier  felt  they  had  a  better  understanding  of 
FBCB2  after  training. 

The  next  question  directly  assessed  avoidance  behaviors  by  stating  “I  don’t  use  FBCB2  if 
I  can  avoid  it.”  Figure  4  shows  the  responses  to  this  statement  before  and  after  training. 
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Figure  4.  Comparison  of  frequencies  of  Soldier  responses  before  and  after  training  for  statement 
7,  “I  don’t  use  reCB2  if  I  can  avoid  it.” 

In  this  case,  the  figure  shows  most  Soldiers  disagree  that  they  avoid  using  FBCB2  both 
before  and  afte- -raining.  Before  training,  four  Soldiers  agreed  (18%  of  those  who  responded) 
whereas  18  di  r.d  (82%)  and  14  did  not  know  or  did  not  respond.  After  training,  29 

disa^eed  (9 1 '  id  only  2  agreed  (9%).  A  higher  percentage  of  the  Soldiers  responded  after 

training  (29  oui  36,  or  81%),  versus  before  training  (only  22  out  of  36,  or  61%). 

The  next  statement  was  also  designed  to  assess  avoidance  behaviors.  The  statement  was 
“FBCB2  really  helps  me  do  my  job.”  It  was  felt  that  Soldiers  who  avoided  using  FBCB2  would 
disagree  with  this  statement.  Figure  5  shows  how  Soldiers  responded  to  this  statement. 

This  figure  shows  the  patterns  of  responses  before  and  after  training  were  roughly  the 
same.  A  high  percentage  of  Soldiers  who  responded  agreed  with  the  statement  before  (96%)  and 
after  training  (93%).  More  Soldiers  strongly  agreed  with  the  statement  after  training  (3  before,  7 
after),  but  also  one  more  Soldier  disagreed  after  training  (1  before,  2  after). 

The  next  statement  was  “FBCB2  isn’t  worth  the  amount  of  effort  you  have  to  put  into  it.” 
This  statement  also  sought  to  assess  avoidance  behaviors,  and  it  was  felt  that  Soldiers  who 
agreed  would  probably  avoid  using  FBCB2.  Responses  to  this  statement  are  shown  in  Figure  6. 
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'FBCB2  really  helps  me  do  my  job" 


(a  pre^raining  DPosHrainin^ 


Figure  5.  Comparison  of  frequencies  of  Soldier  responses  before  and  after  training  for  statement 
8,  “FBCB2  really  helps  me  do  my  job.” 


"FBCB2  isn't  worth  the  amount  of  effort  I  have  to  put  into  it" 


I  a  Pre-training  □  Post-training 


Figure  6.  Comparison  of  frequencies  of  Soldier  responses  before  and  after  training  for  statement 
9,  “FBCB2  isn’t  worth  the  amount  of  effort  you  have  to  put  into  it.” 
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The  figure  shows  that  most  Soldiers  disagreed  with  this  statement  both  before  and  after 
training.  There  were  more  “strongly  disagree”  statements  after  training,  and  fewer  “don’t  know” 
statements. 

Question  10  attempted  to  assess  whether  Soldiers  u,sed  FBCB2  in  a  limited  way. 

Research  from  other  areas  on  automation  use  suggests  some  users  employ  a  few  automated  tools, 
but  do  not  learn  the  entire  range  of  automated  aids  open  to  them.  Soldiers  were  asked  to  respond 
to  the  statement  I  only  use  FBCB2  to  do  a  few  things.”  The  frequencies  of  responses  to  this 
question  are  shown  in  Figure  7. 
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Figure  7.  Comparison  of  frequencies  of  Soldier  responses  before  and  after  training  for  statement 
10,  “I  only  use  FBCB2  to  do  a  few  things.” 

The  figure  indicates  that  prior  to  training  the  majority  of  responses  was  “don’t  know” 
with  the  remaining  responses  leaning  towards  disagreement.  However,  after  training  the 
majority  of  responses  (50%)  strongly  disagreed  with  the  statement  and  another  36%  disagreed. 

In  fact,  there  was  only  one  “agree”  response. 

The  next  several  questions  dealt  with  overuse  behaviors.  Sometimes  users  tend  to  trust 
automation  too  much  (Sarter  &  Woods,  1997)  or  use  it  when  manual  procedures  are  easier  or 
more  appropriate  (Parasuraman  &  Riley,  1997).  Question  1 1  sought  to  assess  how  Soldiers 
viewed  the  accuracy  of  FBCB2  information.  The  statement  was  “I  know  the  information  I  get 
from  FBCB2  is  correct.  The  intent  of  the  statement  was  to  suggest  that  information  from 
FBCBz  is  always  correct.  I  believed  that  Soldiers  who  agreed  with  this  statement  would  tend  to 
place  too  much  trust  in  the  automated  system,  which  has  been  referred  to  as  “automation  bias” 
(Mosier,  Skitka,  Heers,  &  Burdick,  1998).  Figure  8  shows  the  responses  to  this  statement. 


“I  only  use  FBCB2  tc  few  things" 
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As  Figure  8  shows,  most  Soldiers  were  not  sure  about  the  accuracy  of  FBCB2  before 
training,  although  a  number  agreed  that  they  “know”  information  from  FBCB2  is  correct.  After 
training  the  majority  of  Soldiers  disagreed  that  they  “know”  FBCB2  information  is  correct.  This 
suggests  the  training  may  have  given  them  a  more  realistic  view  of  information  accuracy,  and 
consequently  it  may  help  Soldiers  avoid  automation  bias. 


”1  know  that  information  \  get  from  FBCB2  is  correct** 


I B Pre-training  □  Post-training] 
25 


Figure  8.  Comparison  of  frequencies  of  Soldier  responses  before  and  after  training  for  statement 
1 1,  “I  know  the  information  I  get  from  FBCB2  is  correct.” 

Next,  statement  12  was  designed  to  identify  between  avoidance  or  disuse  behaviors.  The 
statement  said,  “If  the  FBCB2  isn’t  working  right,  it’s  best  to  try  to  get  it  working  rather  than 
revert  to  analog.”  I  expected  that  Soldiers  who  disagreed  would  tend  to  avoid  using  digital 
systems,  reasoning  that  manual  (analog)  is  better  and  if  the  system  isn’t  working  they  should 
immediately  revert  to  manual.  Originally,  I  had  hoped  this  question  would  also  identify  overuse 
behaviors,  since  one  of  the  typical  behaviors  of  overuse  is  focusing  on  fixing  a  malfunctioning 
system  at  the  expense  of  performing  the  immediate  task  (Parasuraman  &  Riley,  1997). 

However,  I  realized  that  Soldiers  who  did  not  tend  towards  overuse  could  also  agree  with  the 
statement.  The  responses  are  shown  in  Figure  9. 

The  responses  both  before  and  after  training  are  very  similar,  as  shown  in  the  figure.  The 
main  difference  is  fewer  Soldiers  chose  the  “don’t  know”  response  after  training.  Like  other 
questions,  it  suggests  Soldiers  had  a  better  idea  of  the  parameters  of  the  system  after  training  and 
consequently  expressed  their  opinion  rather  than  choose  “don’t  know.”  Most  Soldiers  (60%) 
agreed  with  the  statement. 
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Statement  13  said,  “FBCB2  is  useful  to  do  a  lot  of  things,  but  I  still  check  to  make  sure 
the  information  it  gives  is  correct.”  Like  statement  1 1  mentioned  above,  this  statement  was 
designed  to  identify  overuse  or  automation  bias  behaviors.  I  expected  that  Soldiers  who 
disagreed  with  the  statement  were  biased  in  favor  of  automation  and  would  show  overconfidence 
behaviors.  A  bar  graph  of  responses  is  shown  in  Figure  10. 

The  figure  sh(  vs  that  the  principle  change  from  pre-training  to  post-training  is  fewer 
Soldiers  chose  the  “don’t  know”  response  after  training.  The  pattern  of  agreement  is  roughly  the 
same,  with  most  Soldier  agreeing  with  the  statement.  Before  training,  63%  of  those  responding 
chose  agree  or  strongly  agree,  while  after  training  80%  chose  these  responses.  However,  on  the 
other  hand,  after  training  1 1%  of  the  Soldiers  chose  to  disagree  with  the  statement.  The  results 
from  this  statement  seem  to  suggest  that  after  training  Soldiers  had  a  better  idea  of  how  far  they 
could  trust  FBCB2  than  before  training. 


"If  the  FBCB2  isn’t  working  right,  It's  best  to  try  to  get  It  working  rather  than  revert  to  analog" 

I M  Pre-training  □  Post- training  | 
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Figure  9.  Comparison  of  frequencies  of  Soldier  responses  before  and  after  training  for  statement 
12,  “If  the  FBCB2  isn’t  working  right,  it’s  best  to  try  to  get  it  working  rather  than  revert  to 
analog.” 


The  last  statement  was  designed  to  measure  Soldier’s  overall  level  of  expertise  with 
FBCB2  and  digitization  in  general.  It  is  based  on  the  idea  that  experts  prefer  shortcuts  and  use 
more  abbreviated  procedures  (Eberts,  1994)  and  when  none  are  available  they  may  develop  their 
own.  The  statement  was,  “Sometimes  there  are  easier  ways  to  do  things  than  what’s  taught  in 
class  or  shown  in  the  book.”  Responses  to  this  statement  are  shown  in  Figure  1 1.  As  the  figure 
shows,  before  training  most  of  the  Soldiers  (83%)  chose  neither  to  agree  or  disagree.  After 
training,  they  were  fairly  evenly  split  between  agreement  and  disagreement. 
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”FBCB2  is  useful  to  do  a  lot  of  things,  but  I  still  check  to  make  sure  the  information  it  gives  is 

correct" 
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Figure  10.  Comparison  of  frequencies  of  Soldier  responses  before  and  after  training  for 
statement  13,  “FBCB2  is  useful  to  do  a  lot  of  things,  but  I  still  check  to  make  sure  the 
information  it  gives  is  correct.” 

Differences  between  pre-  and  post-training  ratings.  To  identify  significant  differences 
between  pre-  and  post-training  responses,  a  Wilcoxon  signed-ranks  test  was  performed  for 
matched  pairs.  The  Wilcoxon  signed-ranks  test  measures  the  direction  of  differences  between 
ratings  of  related  pairs,  but  also  considers  the  magnitude  of  the  differences.  Therefore,  the 
Wilcoxon  is  a  more  powerful  test  than  the  Sign  test  (Siegel,  1956).  Even  so,  response  rates  for 
this  section  were  fairly  low  because  some  respondents  either  did  not  answer  or  chose  the  “don’t 
know”  response.  In  either  case,  these  pairs  of  responses  were  eliminated  from  the  analysis.  This 
accounts  for  the  low  n  of  most  of  the  statements  in  this  section. 

Two  statements,  10  and  1 1,  showed  significant  differences.  Statement  10  “I  only  use 
FBCB2  to  do  a  few  things,”  showed  significantly  less  agreement  with  the  statement  after  training 
(z  =  -3.22,  p<.001),  which  suggests  Soldiers  were  willing  to  use  FBCB2  for  more  tasks  or  use 
more  of  the  tools,  or  both. 

In  addition,  statement  11,  “The  information  on  FBCB2  is  almost  always  correct,  so  I 
don’t  bother  checking  it,”  also  showed  significantly  less  agreement  after  training  (z  =  -3.464,  p< 
.001).  This  could  mean  that  Soldiers  had  a  better  appreciation  of  the  accuracy  of  digital 
information,  or  at  least  of  the  need  to  cross-check  information.  Since  failing  to  cross-check 
information  from  automated  systems  is  a  behavior  associated  with  over  use  or  over  reliance  on 
automation,  the  training  may  reduce  the  chances  of  Soldiers  engaging  in  these  behaviors. 

Statement  six,  “Sometimes  the  FBCB2  does  things  for  no  reason  or  that  I  don’t 
understand,”  (see  Figure  3)  had  relatively  few  usable  responses  (n  =  15).  The  low  n  is  primarily 
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a  result  of  “don’t  know”  responses  which  were  not  counted  in  this  analysis.  Although  the  results 
do  not  meet  the  error  probability  criteria  of  p.<  .05  (z  =  -1.82,  n.s.),  an  examination  of  Figure  3 
shows  there  were  more  disagreement  responses  after  training  than  before,  which  suggested  there 
may  be  a  difference  between  pre-  and  post-training  responses  which  is  not  evident,  perhaps  due 
to  an  intervening  variable.  Our  search  for  this  variable  revealed  that  Soldiers  who  had  previous 
FBCB2  experience  responded  differently  from  those  with  no  previous  experience.  When  I 
anal^•7ed  each  of  these  groups  independently,  I  found  Soldiers  with  previous  FBCB2  experience 
sho\.  d  a  significant  difference  in  responses  before  and  after  training  (z  =  -2.43,  p  <  .015). 

These  Soldiers  reported  they  had  a  better  under,'-  anding  of  FBCB2  after  training. 

*  ?ior  etimes  there  are  easier  ways  to  do  things  on  FBCB2  than  what's  taught  In  class  or 

shown  in  the  book" 


[m  Pre-trainir  • '  ”ost-train»ng  j 


Figure  11.  Comparison  of  frequencies  of  Soldier  responses  before  and  after  training  for 
statement  14,  “Sometimes  there  are  easier  ways  to  do  things  than  what’s  taught  in  class  or  shown 
in  the  book.” 

Statement  14,  “Sometimes  there  are  easier  ways  to  do  things  on  FBCB2  than  what’s 
taught  in  class  or  shown  in  the  book”  (see  Figure  11),  also  has  a  low  response  rate  («  =  15)  and 
does  not  meet  the  .05  confidence  level  (z  =  -1.63,  n.s.). 

Questionnaire  Section  Three:  FBCB2  Monitoring  Behavior 

Section  three  consisted  of  two  questions.  For  each  question  Soldiers  were  asked  to  select 
from  a  series  of  responses.  The  first  question  was,  “How  often  would  you  look  at  the  FBCB2 
display  after  you  crossed  the  Line  of  Departure  (LD)?”  The  purpose  of  this  question  was  to 
gather  data  on  how  Soldiers  employed  the  FBCB2  display  during  movement.  The  possible 
responses  were  (a)  “Never,”  (b)  “Every  few  minutes,”  (c)  “Every  few  seconds,”  (d)  “If  I’m 
lost/confused  or  when  approaching  the  next  decision  point,”  and  (e)  “Other.”  Soldiers  responses 
are  shown  in  Figure  12. 
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"How  often  would  you  look  at  the  FBCB2  display  after  crossing  the  Line  of  Departure?" 
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Figure  12.  Comparison  of  frequencies  of  Soldier  responses  before  and  after  training  for  question 
15,  “How  often  would  you  look  at  the  FBCB2  display  after  you  crossed  the  Line  of  Departure 
(LD)?” 

The  bar  graph  shows  that  the  pattern  of  Soldier  responses  changed  little  from  before  to 
after  training.  Most  Soldiers  said  they  would  check  the  FBCB2  display  every  few  minutes  after 
crossing  the  LD.  After  training,  fewer  Soldiers  said  they  would  never  check  the  display,  and 
more  said  they  would  check  it  every  few  seconds.  Also,  after  training  more  Soldiers  admitted 
they  would  check  the  display  if  they  were  lost  or  confused,  or  approaching  a  waypoint.  There 
were  also  fewer  “other”  answers  after  training.  Most  of  the  Soldiers  who  marked  “other”  before 
training  added  that  they  didn’t  know  how  often  they  would  monitor  the  display. 

The  second  question  in  this  section  asked  about  sharing  tasks  with  others,  often  known  as 
“task  shedding”  (Sirevaag,  Kramer,  Wickens  Sc  Reisweber,  1993).  Specifically,  I  wanted  to 
know  how  Soldiers  handled  the  requirement  to  monitor  FBCB2.  The  question  asked,  “How 
often  would  you  hand-off  the  requirement  to  monitor  the  FBCB2  display  to  someone  else?” 
Soldiers  could  choose  (a)  “Never,”  (b)  “If  I’m  doing  something  else,”  (c)  “Every  few  minutes,” 
(d)  “Always,”  or  (e)  “Other.”  The  responses  to  this  question  are  shown  in  Figure  13. 

Like  the  previous  question,  the  pattern  of  responses  was  very  similar  both  before  and 
after  training.  Most  Soldiers  agreed  they  would  delegate  the  monitoring  task  if  they  were 
involved  with  another  task.  A  few  Soldiers  before  training  said  they  would  “always”  delegate 
FBCB2  monitoring,  whereas  after  training  no  one  chose  this  response.  The  “other”  responses 
after  training  mostly  fell  into  the  “Don’t  know,”  or  “Depends  on  the  situation”  category. 
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**How  often  would  you  hand-of  the  requirement  to  monitor  FBCB2  to  someone  else?” 


Never  If  Doing  Something  Else  Every  Few  Minutes  Always  Other 


Figure  1 3.  Comparison  of  frequencies  of  Soldier  responses  before  and  after  training  for  question 
16, “How  often  would  you  hand-off  the  requirement  to  monitor  the  FBCB2  display  to  someone 
else?” 


Pre-  post-training  differences.  I  also  wanted  to  know  how  many  Soldiers  changed  their 
answers  after  training,  and  if  so,  if  the  changes  formed  a  pattern.  To  answer  these  questions,  I 
examined  Soldier  responses  before  and  after  training  and  developed  a  matrix  of  all  possible 
combinations  of  responses.  I  then  counted  the  frequency  of  response  for  each  possible 
combination.  I  identified  how  many  Soldiers  had  the  same  response  pre-  and  post-training  versus 
how  many  changed  their  responses.  For  those  cases  where  Soldiers  changed  their  responses,  I 
performed  a  chi-square  test  to  determine  if  any  response  pattern  occurred  significantly  more 
often  than  the  others. 

For  question  15,  20  out  of  35  Soldiers  (57%)  had  similar  responses  pre-  and  post-training. 
Of  those  who  changed  their  responses,  there  was  no  discemable  pattern  (x^  [7,  A^=15]  =  7.93, 
W.5.).  Question  16  had  24  Soldiers  make  the  same  response  (67%)  while  12  changed  responses. 
The  changed  responses  had  no  significant  pattern  for  this  question  either  (x^  [6,  A^=12]  =  3.17, 
n.s.). 

Questionnaire  Section  Four:  Free  Response  Section 

The  last  section  had  two  free  response  questions.  The  first  question  asked  for  more 
information  about  task  shedding  the  requirement  to  monitor  FBCB2.  The  question  asked,  “If 
you  wanted  to  hand-off  monitoring  the  FBCB2  display  to  someone  else,  who  would  you  hand  it 
off  to?”  There  were  a  number  of  different  responses  to  this  question,  although  responses  tended 
to  fall  into  several  categories.  Table  9  shows  the  major  categories  of  responses  and  the 
frequencies  of  response  in  each  one.  As  the  table  shows,  the  most  popular  answer  to  this 
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question  before  and  after  training  was  either  hand  it  off  to  another  crew  member  (in  a  vehicle 
crew)  or  to  the  commander  or  leader.  Before  training,  several  respondents  said  they  didn’t  know 
who  they  would  hand  the  monitoring  task  to.  Probably  the  most  interesting  finding  from  this 
question  is  that  after  training  a  number  of  respondents  said  they  would  hand  it  off  to  a  trained 
user.  This  suggests  that  after  training  Soldiers  realized  the  need  to  have  FBCB2  operated  by 
trained  users. 

The  second  question  in  this  section  asked,  “What  feature  of  this  FBCB2  is  the  most 
difficult  to  work  with?”  Again,  there  were  a  number  of  responses  which  tended  to  fall  into 
categories.  Before  training  most  Soldiers  reported  they  didn’t  know,  which  suggests  they  had 
little  experience  with  the  system  before  training,  while  a  few  reported  using  the  touch  screen  in 
rough  terrain  or  while  wearing  chemical  protective  gear  was  difficult.  After  training,  Soldiers 
had  a  better  idea  of  how  difficult  tasks  were. 

Table  9.  Frequencies  Of  Soldier  Responses  To  Question  17,  “If  You  Wanted  To  Hand-Off 
Monitoring  The  FBCB2  Display  To  Someone  Else,  Who  Would  You  Hand  It  Off  To?” 


Response  category 

Frequency 

Pre-training 

Post-training 

Don’t  know 

3 

0 

Another  crew  member 

12 

9 

Commander/leader 

10 

9 

An  assistant 

2 

5 

A  trained  user 

0 

4 

The  two  most  frequent  response  categories  after  training  involved  using  overlays  and 
working  with  messages  and  reports.  Somewhat  surprisingly,  after  training  several  Soldiers 
reported  they  found  all  features  of  the  FBCB2  easy  to  use.  Response  frequencies  for  this 
question  are  sununarized  in  Table  10. 

Table  10.  Frequencies  Of  Soldier  Responses  To  Question  18,  “What  Feature  Of  The  FBCB2  Is 
The  Most  Difficult  To  Work  With?” 


Response  category 

Frequency 

Pre-training 

Post-training 

Don’t  know 

22 

2 

Using  the  touch  screen 

3 

1 

0 

5 

Planning 

1 

2 

Using  overlays 

2 

8 

No  features  are  difficult 

0 

4 
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Discussion 


As  mentioned  in  the  introduction,  there  were  a  number  of  objectives  to  this  research.  The 
first  was  to  determine  how  training  affected  Soldiers  attitudes  and  behaviors  towards  digital 
systems,  and  also  how  it  changed  their  use  of  the  systems.  The  second  was  to  identify  what 
capabilities  of  digital  systems,  in  this  case  FBCB2,  Soldiers  found  more  or  less  useful.  The  last 
objective  was  to  identify  if  Soldiers  exhibited  certain  behavior  patterns  towards  digital  systems. 

I  will  discuss  how  this  research  addressed  each  of  these  objectives. 

The  Effect  of  Training 

Formal  training  seems  to  have  a  significant  and  generally  positive  effect  on  Soldier 
attitudes  and  behaviors  towards  digitization.  After  training,  Soldiers  reported  they  felt  they 
understood  the  system  better,  they  were  more  likely  to  use  it,  and  used  more  of  the  features.  This 
supports  the  idea  that  Soldiers  who  understand  how  to  use  digital  tools  are  more  likely  to  employ 
them  Soldiers  also  reported  they  better  understood  FBCB2  weaknesses  after  training. 

There  was  a  statistically  significant  difference  between  pre-training  and  post-training 
aUitudes  and  behaviors.  Of  the  five  questions  in  section  one,  four  of  them  had  rankings  which 
differed  significantly  from  pre-  to  post-training.  Only  question  one;  “What  do  you  consider  the 
most  useful  planning  capabilities  of  FBCB2”  showed  no  significant  difference.  When  Soldiers 
were  asked  to  rank  order  the  usefulness  of  FBCB2  capabilities,  there  was  more  agreement  among 
Soldiers  after  training  than  before  for  each  question.  This  suggests  the  training  gives  Soldiers  a 
better  appreciation  of  FBCB2  capabilities.  When  asked  when  FBCB2  was  most  useful,  the 
responses  “planning”  and  “preparation  for  combat”  were  rated  significantly  higher  after  training, 
so  it  may  be  that  Soldiers  better  understood  these  concepts  after  training. 

One  point  that  could  be  raised  is  that  40  hours  of  computer  experience  alone  could  have 
affected  attitudes  and  behaviors  independent  of  material  presented  in  class.  While  this  is 
possible,  some  evidence  from  this  research  contradicts  this  point.  Some  of  the  questions  I  asked 
concerned  topics  not  covered  in  the  class  material.  These  topics  had  less  change,  or  no  change, 
from  pre-  to  post-training.  For  example,  questions  1 5  asked  about  monitoring  FBCB2  after 
crossing  the  LD,  and  question  16  asked  about  how  Soldiers  would  hand-off  monitoring 
requirements.  Neither  of  these  topics  are  covered  in  OPNET,  and  neither  of  these  questions 
showed  significant  differences  between  the  pre-test  and  post-test.  This  suggests  that  it  is  the 
material  presented  in  class  which  changes  Soldiers’  attitudes  and  behaviors,  rather  than  simple 
experience  working  with  digital  systems.  Therefore,  structured,  formal  training  may  have  a  more 
significant  effect  than  unstructured  experience  with  the  system. 

Soldier  Ratings  of  Usefulness 

Soldiers  generally  feel  FBCB2  is  useful,  and  is  worth  the  additional  effort  required  to 
employ  digitization.  They  reported  it  is  most  useful  for  planning,  preparation  for  combat,  and 
movement,  but  less  during  combat  (i.e.  enemy  contact).  Once  maneuver  units  are  in  contact  with 
the  enemy,  they  are  focused  on  the  enemy  and  typically  do  not  have  time  to  view  digital 
displays.  Also,  once  the  enemy  is  seen,  the  enemy  in  range  has  higher  priority  than  any  enemy 
displayed  on  digital  systems  which  is  out  of  range.  These  phases  (planning,  preparation  for 
combat,  and  movement)  also  were  reported  to  have  the  highest  workload.  This  naturally  follows 
since  those  features  that  are  used  least  will  not  have  high  workloads  associated  with  them. 
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Soldiers  ranked  the  most  useful  planning  capabilities  as  sending  and  receiving  messages 
and  overlays,  and  route  planning.  The  most  useful  non-planning  related  features  ranked  as  most 
useful  were  sending  and  receiving  messages,  navigating  to  a  location,  checking  the  location  of 
enemy  units,  and  avoiding  threats  such  as  minefields.  Overall,  Soldiers  reported  that  those 
features  of  FBCB2  which  had  the  highest  workload  and  were  most  difficult  to  work  with  were 
also  those  rated  the  most  useful. 

Soldiers  did  not  agree  on  the  accuracy  of  different  features  of  FBCB2  either  before  or 
after  training.  Although  agreement  was  higher  after  training,  it  was  still  fairly  low  (Kendall’s  W 
=  .161  after  training).  This  is  probably  because  accuracy  was  not  covered  in  training  and 
Soldiers  may  not  have  had  enough  experience  with  the  system  to  estimate  accuracy. 

Soldier  Patterns  of  Behavior  Towards  Digitization 

One  of  the  goals  of  this  research  was  to  see  if  Soldiers  exhibited  behaviors  towards 
digital  systems  that  were  similar  to  behaviors  exhibited  by  people  in  general  towards  automated 
systems.  The  behaviors  I  was  interested  in  were  disuse  or  avoidance,  limited  use,  overuse  or 
over-reliance  on  automation/digitization,  and  expert  use.  The  second  section  of  the  questionnaire 
asked  questions  which  were  designed  to  measure  these  behavior  patterns. 

Previous  observations  of  Soldiers  and  leaders  using  FBCB2  for  exercises  showed 
considerable  disuse  behaviors.  However,  Soldiers  in  this  investigation  reported  few  disuse 
behaviors,  and  the  majority  reported  a  willingness  to  employ  FBCB2  for  the  majority  of  combat 
tasks.  This  apparent  disparity  has  several  possible  explanations.  First,  the  Soldiers  in  this 
investigation  were  undergoing  formal  training  for  the  digital  system,  whereas  Soldiers  in  the 
observational  research  reported  much  of  their  training  was  informal  OJT.  Soldiers  in  formal 
training  may  have  felt  more  committed  to  using  digital  systems  since  they  were  expending  effort 
to  learn  to  use  the  systems.  Second,  the  Soldiers  in  the  observational  research  effort  were  from  a 
different  unit  from  the  Soldiers  in  the  present  investigation.  The  different  units  may  have  had 
different  command  emphasis  on  the  value  of  digitization.  Since  commanders  and  leaders  often 
have  a  significant  impact  on  the  organizational  climate  of  a  unit  (Bass,  1981),  different  command 
emphasis  on  digitization  could  have  been  reflected  in  the  Soldier’s  willingness  to  use  digital 
systems  in  the  present  effort. 

One  question  attempted  to  identify  limited  use  behaviors,  “I  only  use  FBCB2  to  do  a  few 
things.”  Although  there  were  a  fair  number  of  agreement  responses  to  this  question  before 
training,  after  training  the  majority  of  responses  disagreed  or  strongly  disagreed.  This  suggests 
these  Soldiers  have  moved  beyond  limited  use  and  are  willing  to  employ  digital  systems  for 
many  combat  tasks. 

I  also  wanted  to  see  if  Soldiers  exhibited  overuse  or  over-reliance  in  digital  systems. 

Two  questions  assessed  this  behavior,  “I  know  the  information  I  get  from  FBCB2  is  correct,” 
and  “FBCB2  is  useful  to  do  a  lot  of  things,  but  I  still  check  to  make  sure  the  information  is 
correct.”  Both  of  these  questions  dealt  with  cross-checking  the  accuracy  of  information  from  the 
digital  system,  because  one  of  the  most  observable  behaviors  associated  with  over-reliance  in 
automation  is  automation  bias,  that  is,  assuming  the  information  from  the  automated  system  is 
correct  without  checking  it.  The  answers  to  both  of  the  questions  indicate  that  Soldiers  are  not 
biased  in  favor  of  automation  and  they  report  they  check  the  information  they  get  from  FBCB2. 


23 


The  last  behavior  set  I  was  interested  in  was  expert  behaviors.  In  fact,  there  is  relatively 
little  information  about  what  constitutes  general  expert  behavior  with  computer  systems.  Most 
expert  behavior  measures  involve  speed  and  accuracy  with  particular  software  applications,  but 
not  expert  computer  use  in  general.  One  behavior  that  is  thought  to  indicate  expertise  is  that 
experts  tend  to  take  advantage  of  “hot  keys”  and  shortcuts  more  often  than  do  others  (Eberts, 
1994),  and  frequently  will  develop  their  own  shortcuts  for  tasks.  Therefore,  1  asked  Soldiers  to 
agree  or  disagree  with  the  statement,  “Sometimes  there  are  e.  :;ier  ways  to  do  things  on  FBCB2 
than  what’s  taught  in  class  or  shown  in  the  book.”  I  expected  that  Soldiers  who  were  more 
expert  and  tended  to  use  shortcuts  would  agree  with  this  statement. 

The  results  showed  that  before  training  most  Soldiers  chose  the  “don’t  know”  option,  but 
after  training  there  was  an  almost  even  split  between  agree  and  disagree  with  the  statement. 

When  the  “don’t  know”  responses  are  eliminated,  48%  of  the  Soldiers  agreed  with  the  statement 
after  training,  which  suggests  to  us  that  about  half  of  the  Soldiers  feel  comfortable  enough 
working  with  FBCB2  to  use  shortcuts.  Somewhat  surprisingly,  there  were  no  correlations  found 
between  this  variable  and  previous  computer  experience  (or  indeed  any  other  demographic). 

In  general.  Soldiers  in  this  investigation  did  not  exhibit  any  of  the  maladaptive  patterns  of 
behavior  seen  in  other  areas,  nor  indeed  observed  in  other  digital  units.  Given  that  this  is  a  field 
effort  with  many  extraneous  variables  beyond  our  control,  it  is  best  to  be  cautious  in  drawing 
conclusions.  However,  several  of  the  statements  which  attempted  to  assess  maladaptive 
behaviors  showed  that  Soldiers’  responses  showed  less  of  these  behaviors  after  training  than 
before.  Of  particular  note  is  that  Soldiers  learned  to  take  information  they  got  from  FBCB2  with 
caution,  thus  avoiding  one  of  the  major  problems  of  automation  overuse. 

Teamwork  Issues 

An  additional  set  of  questions  was  asked  which  dealt  with  how  Soldiers  would  hand-off 
monitoring  of  the  digital  system.  The  majority  of  Soldiers  said  they  would  hand-off  monitoring 
requirements  if  they  were  doing  something  else,  and  they  would  hand  it  off  to  another  crew 
member  or  their  commander.  The  responses  were  fairly  similar  both  before  and  after  training, 
which  may  be  because  this  topic  is  not  covered  in  the  classes. 

Methodological  Considerations 

There  are  several  caveats  related  to  using  the  results  of  reported  herein.  The  first  is  that 
there  was  a  somewhat  restricted  range  since  only  Soldiers  E5  and  above  were  given  the 
questionnaires.  This  means  the  results  may  not  be  representative  of  the  views  of  lower-level 
enlisted  Soldiers. 

The  second  consideration  is  that  the  partieipant  group  contained  Soldiers  who  may  have 
used  digital  systems  (FBCB2)  in  different  ways.  Some  Soldiers  in  the  participant  group 
belonged  to  maneuver  units,  typically  platoons  or  companies,  while  others  operated  FBCB2  in 
TOCs.  Soldiers  in  platoons  and  companies  typically  would  have  FBCB2  in  their  combat 
vehicles  and  would  use  them  to  move  and  engage  the  enemy.  On  the  other  hand.  Soldiers  in 
TOCs  typically  would  use  FBCB2  to  monitor  and  support  combat.  Therefore,  the  way  Soldiers 
used  FBCB2  in  maneuver  units  versus  in  TOCs  may  be  significantly  different.  This  suggests 
that  this  could  have  been  a  confounding  variable  which  affected  how  Soldiers  answered  the 
questions.  This  could  explain  in  part  the  low  agreement  ratings  (Kendall’s  W)  and  multi- 
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modalities  of  some  of  the  questions.  In  the  future  I  intend  to  include  a  question  asking  Soldiers 
whether  they  use  the  digital  system  in  a  maneuver  unit  or  in  a  TOC. 

Future  Research 

It  might  be  instructive  to  perform  similar  research  on  digital  systems  other  than  FBCB2 
and  identify  similarities  and  differences.  In  addition,  future  research  might  involve  a  more  in- 
depth  investigation  of  how  Soldiers  employ  FBCB2  functions  and  tools,  and  of  its  perceived 
utility.  Another  variable  which  might  be  included  in  future  research  is  whether  Soldiers  use 
FBCB2  in  maneuver  units  or  TOCs.  As  mentioned  above.  Soldiers  may  utilize  FBCB2 
differently  in  these  different  environments,  and  an  understanding  of  these  difference  may  have  an 
impact  on  training. 

Summary  of  Findings  and  Recommendations 

Formal  training.  Formal  training  seems  to  have  a  significant  impact  on  Soldier  attitudes 
and  behaviors  towards  digital  systems.  Classes  on  digital  systems  seem  to  give  Soldiers  a 
positive  attitude  towards  digital  systems  or  reinforce  a  previous  positive  attitude.  Soldiers  report 
they  better  understand  the  system  and  tend  to  use  it  more  and  use  more  features.  Training 
introduces  Soldiers  to  features  of  digital  systems  which  are  useful  to  them  that  they  might  not 
otherwise  learn  through  discovery  learning  or  OJT.  In  addition,  training  seems  to  steer  Soldiers 
away  from  maladaptive  behavior  patterns  like  those  seen  with  automation  in  other  areas. 

Soldiers  are  willing  to  use  digital  systems  and  tools,  but  at  the  same  time  avoid  placing  too  much 
confidence  in  the  system. 

FBCB2  usability.  Future  design  improvements  for  the  FBCB2  interface  should  focus  on 
improving  the  usability  of  messages  and  reports,  and  also  the  use  of  overlays.  These  are  the 
functions  Soldiers  report  they  use  most,  and  are  also  the  most  difficult  to  use  and  require  the 
highest  workload. 

Summary 

This  research  shows  that  formal  training  in  digital  systems  has  a  significant  positive 
affect  on  Soldiers’  attitudes  and  behaviors  towards  digital  systems.  Not  only  does  formal 
training  better  prepare  Soldiers  to  use  correct  procedures  to  produce  digital  products,  it  also  gives 
Soldiers  more  confidence  in  their  ability  to  employ  digitization.  Soldiers  who  complete  formal 
OPNET  courses  report  they  are  more  likely  to  use  digitization  and  use  more  of  the  tools 
available.  In  addition.  Soldiers  seem  to  have  a  better  appreciation  of  digitization’s  strengths  and 
weaknesses  after  training. 

The  effort  also  sought  to  find  out  if  Soldiers  exhibited  certain  maladaptive  behaviors 
towards  digitization  similar  to  those  seen  in  other  domriins,  such  as  disuse  or  avoidance,  limited 
use,  or  overuse.  I  found  that  this  sample  of  Soldiers  exhibited  few  of  these  behaviors  before 
training,  and  even  less  after  training.  In  addition,  I  asked  these  Soldiers  about  behaviors  thought 
to  be  associated  with  computer  expertise.  At  the  end  of  training,  about  39%  of  Soldiers  in  this 
sample  reported  they  exhibited  these  behaviors,  although  there  was  no  correlation  with  previous 
computer  experience. 


25 


26 


References 


Barnett,  J.  S,,  Metiza,  L.  L.  (2(X)3).  AuUnnatkm  mtegralhm:  Contpaniig  flight  deck  atilomalion 
and  ILS.  Army  digitizatJan.  Paper  t<.)  be  presented  at  tltc  Interscrvice/lnduslry  1' raining, 
Simulaiion.  and  Erlucation  Conference  (I/ITSECK  December  ]-4,  2(K)3. 

Barnett,  J,  S.,  Meliza,  L.  L.  &  McCluskcy,  M,  R.  (2001 ),  Defining  digital prqficiemy 

measurement  targets  for  IJ.S,  Army  units  (AR!  technical  report  1 1 17).  Alexandria.  V.A: 
Army  Research  Institute  for  the  Behavioral  and  .Social  Sciences. 

Bass,  B.  M.  (1981).  StogdiU's  handbook  of  leadership:  A  survey  of  theory  and  research.  New 
York:  The  Free  Press. 

Eberts,  R.  E.  (1994)  User  interface  design.  Englewood  Cliffs,  NJ:  Prentice  Hall. 

Mo.sier,  K.  L.,  Skitka,  1,.  .1.,  Heers,  S.  &  Burdick,  M.  (1998).  Automation  bias:  Deci.sion  making 
and  performance  in  high-tech  cockpits.  International  Journal  of  Aviation  Psychology  8 
(1).  47-6-3. 

Parasuraraan,  R.  &  Riley,  V.  (1997).  Human  and  automation;  Use,  misuse,  disu.se  and  abuse. 
Human  Factors  39  (2),  230-253. 

Sarter,  N.  B.  &  Woods,  D.  D.  (1997).  Team  play  with  a  powerful  and  independent  agent; 

Operational  experiences  and  automation  surpri.ses  with  the  Airbus  A-20.  Human  Factors 
39  (4)  553-569. 

Siegel,  S.  (1956).  Nonparametric  statistics  for  the  social  sciences.  New  York:  McGraw  Hill. 

Sirevaag,  E.  T,  Kramer,  A.  F.,  Wickens,  C.  D.  &  Reisweber,  M.  (1993).  Assessment  of  pilot 

l^rformance  and  mental  workload  in  rotary  wing  aircraft.  Ergonomics  36  (9),  1 121-1 140. 

Tabachnick,  B.  G.  &  Fidell,  L.  S.  (1996).  Using  multivariate  statistic.s,  3*^^  Ed.  New  York;  Harper 
Collins. 


27 


28 


Appendix  A 

Acronyms 

BCTC.  Battle  Command  Training  Center 
BOS.  Battlefield  Operating  System. 

FBCB2.  Force  XXI  Battle  Command,  Brigade  and  Below. 

GPS.  Global  Positioning  System. 

LAN.  Local  Area  Network. 

LD.  Line  of  Departure. 

MOS.  Military  Occupational  Specialty. 

OC.  Observer-Controller. 

OJT.  On-the-job-training. 

OPNET.  Operator  new  equipment  training 

PEOSTRI.  Program  Executive  Officer,  Simulation,  Training,  and  Instrumentation. 
SOP.  Standard  Operating  Procedure. 

TOC.  Tactical  Operations  Center. 

TTP.  Tactics,  Techniques,  and  Procedures. 
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Appendix  B 

Pre-Training  Questionnaire 


B-1 


B-2 


PARTICIPANT  # 


FT  60-49A 


Not  to  be  shown  to  unauthorized  persons.  Not  to  be  reproduced  without  the  specific  permission  of  the 
Technical  Director,  U.S.  ARMY  RESEARCH  INSTITUTE  FOR  THE  BEHAVIORAL  AND  SOCIAL  SCIENCES 


QUESTIONS  ABOUT  YOU 


(A)  Your  Rank: _ (B)  Your  MOS: _ 

(C)  How  long  have  you  been  in  the  Army  (total)  Years _ Months _ 

(D)  What  digital  system  do  you  normally  work  with?  (circle  all  that  apply) 

FBCB2  ASAS  AFATDS  AMDWS  MCS/ (Lite)  CSSCS  Other 

(E)  How  many  months  have  you  been  using  FBCB2? _ 

(F)  About  how  many  hours  a  month  do  you  use  FBCB2? _ _ 

(G)  Have  you  ever  taken  an  on-line  tutorial  for  FBCB2?  YES  NO 

Please  continue  to  the  next  page 
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Section  1 

For  each  of  the  following  questions,  please  rank  order  the  list  of  items  from  first  to  last. 


1 .  What  do  you  consider  the  most  useful  planning  capabilities  with  FBCB2?  Please  rank  order  from  most 
useful  (number  1)  to  least  useful 

_ Plan  a  route  _ Plan  support  measures 

_ Send/receive  messages/plans  /overlays  _ Line-of-sight  tools 

_ Other _  _ Other _ _ 

2.  What  do  you  consider  the  most  useful  FBCB2  capabilities  other  than  planning?  Please  rank  order  from 
most  useful  (number  1)  to  least  useful 

_ Send/receive  messages 


.  Navigate  to  a  location 
.  Avoid  threats  (minefields,  NBC  areas,  etc.) 
.  Check  location  of  subordinate  units 
Check  status  of  subordinate  units 


.  Logistics/personnel  reporting 
.Check  location  of  enemy  units 
Other _ 


3.  When  is  FBCB2  most  useful  to  you  ?  Rank  order  from  most  useful  (number  1)  to  least  useful 

_ Planning  _ Assembly  area  operations 

_ Preparation  for  combat  _ Movement 

_ Attack  _ Actions  on  the  objective 

_ Defense  _ _ Other _ 


4.  When  is  FBCB2  workload  the  highest?  Rank  order  from  most  work  (number  1 )  to  least  work 

_ Planning  _ Assembly  area  operations 

_ Preparation  for  combat  _ Movement 

_ Attack  _ Actions  on  the  objective 

_ Defense  _ Other _ 


5.  On  the  FBCB2  display,  which  of  the  following  is  usually  CLOSEST  to  ground  truth  (most  accurate)? 
Please  rank  order  from  most  accurate  (number  1)  to  least  accurate 

_ Friendly  icons  (center  of  mass)  _ Friendly  icons  (individual  platforms) 

_ Line-of-sight  analysis  _ Enemy  icons 

_ Terrain  features  _ Overlays 

_ Other _ 

Please  continue  to  the  next  page 
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Section  2 

•  For  each  of  the  following  statements,  circle  whether  you  STRONGLY  AGREE,  AGREE,  DISAGREE,  or 
STRONGLY  DISAGREE  with  the  statements,  or  circle  DON’T  KNOW  if  you’re  not  sure. _ 

6.  “Sometimes  the  FBCB2  does  things  for  no  reason,  or  that  I  don’t  understand" 

Don’t  Strongly  Disagree  Agree  Strongly 

Know  Disagree  Agree 

7.  “I  don’t  use  FBCB2  if  I  can  avoid  it” 


Don’t  Strongly  Disagree  Agree 

Know  Disagree 

8.  “FBCB2  really  helps  me  do  my  job” 

Don’t  Strongly  Disagree  Agree 

Know  Disagree 


Strongly 

Agree 


Strongly 

Agree 


9.  “FBCB2  isn’t  worth  the  amount  of  effort  I  have  to  put  into  if 

Don’t  Strongly  Disagree  Agree 

Know  Disagree 

10.  “I  only  use  FBCB2  to  do  a  few  things" 

Don’t  Strongly  Disagree  Agree 

Know  Disagree 


Strongly 

Agree 


Strongly 

Agree 


1 1 .  “I  know  that  information  I  get  from  FBCB2  is  correcf 

Don’t  Strongly  Disagree  Agree 

Know  Disagree 


Strongly 

Agree 


12.  “If  the  FBCB2  isn’t  working  right,  it’s  best  to  try  to  get  it  working  rather  than  revert  to  analog" 


Don’t  Strongly  Disagree 

Know  Disagree 


Agree  Strongly 

Agree 


Please  continue  to  the  next  page 
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1 3.  “FBCB2  is  useful  to  do  a  lot  of  things,  but  I  still  check  to  make  sure  the  information  it  gives  is  correcf 


Don’t  Strongly  Disagree  Agree  Strongly 

Know  Disagree  Agree 

14.  “Sometimes  there  are  easier  ways  to  do  things  on  FBCB2  than  what’s  taught  in  class  or  shown  in  the 
book” 


Don’t 

Strongly 

Disagree 

Agree 

Strongly 

Know 

Disagree 

Agree 

Section  3 

•  For  each  of  the  following  questions,  please  circle  your  answer. 


15.  How  often  would  you  look  at  the  FBCB2  display  after  you  crossed  the  Line  of  Departure  (LD)?  (Circle) 

If  I’m  lost/confused  or  when  approaching  the  next  decision  point  Never 

Every  few  minutes  Every  few  seconds  Other _ 

16.  How  often  would  you  hand-off  the  requirement  to  monitor  the  FBCB2  display  to  someone  else?  (circle 
one) 

Never  If  I’m  doing  something  else  Every  few  minutes 

Always  Other _ 


Section  4 

•  Please  answer  the  questions  by  filling  in  the  blank. 


17.  If  you  wanted  to  hand-off  monitoring  the  FBCB2  display  to  someone  else,  who  would  you  hand  it  off  to? 


18.  What  feature  of  this  FBCB2  is  the  most  difficult  to  work  with? 


You’re  done!  Return  this  questionnaire  to  the  envelope.  Thanks! 
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Appendix  C 

Post-Training  Questionnaire 
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C-2 


PARTICIPANT# 


FT  60-49B 


Not  to  be  shown  to  unauthorized  persons.  Not  to  be  reproduced  without  the  specific  permission  of  the 
Technical  Director,  U.S.  ARMY  RESEARCH  INSTITUTE  FOR  THE  BEHAVIORAL  AND  SOCIAL  SCIENCES 


Questions  About  You 


(A)  About  how  many  hours  a  week  do  you  use  a  computer  at  work? _ 

at  home? _ 

(B)  About  how  many  hours  a  week  do  you  play  computer  games? _ 

(C)  If  you  play  computer  games,  which  ones  do  you  play  (circle  all  that  apply) 


Card  games 

Strategy  games 

Arcade-type  games 

Role-playing  games 

Simulation  games 

Sports  games 

Other 

Other 

Please  continue  to  the  next  page 
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Section  1 

For  each  of  the  following  questions,  please  rank  order  the  list  of  items  from  first  to  last. 


1 .  What  do  you  consider  the  most  useful  planning  i 
useful  (number  1 )  to  least  useful 

_ Plan  a  route 

_ Send/receive  messages/plans  /overlays 

_ Other _ 


with  FBCB2?  Please  rank  order  from  most 

_ Plan  support  measures 

_ Line-of-sight  tools 


2.  What  do  you  consider  the  most  useful  FBCB2  capabilities  other  than  planning?  Please  rank  order  from 
most  useful  (number  1)  to  least  useful 

_ Navigate  to  a  location  _ Send/receive  messages 

_ Avoid  threats  (minefields,  NBC  areas,  etc.)  _ Logistics/personnel  reporting 

_ Check  location  of  subordinate  units  _ Check  location  of  enemy  units 

_ Check  status  of  subordinate  units  _ Other _ 

3.  When  is  FBCB2  most  useful  to  you  ?  Rank  order  from  most  useful  (number  1)  to  least  useful 

_ Planning  _ Assembly  area  operations 

_ Preparation  for  combat  _ Movement 

_ Attack  _ Actions  on  the  objective 

_ Defense  _ Other _ 

4.  When  is  FBCB2  workload  the  highest?  Rank  order  from  most  work  (number  1)  to  least  work 

_ Planning  _ Assembly  area  operations 

_ Preparation  for  combat  _ Movement 

_ Attack  _ Actions  on  the  objective 

_ Defense  _ Other _ 


5.  On  the  FBCB2  display,  which  of  the  following  is  usually  CLOSEST  to  ground  truth  (most  accurate)? 
Please  rank  order  from  most  accurate  (number  1)  to  least  accurate 


.  Friendly  icons  (center  of  mass) 

.  Line-of-sight  analysis 
.  Terrain  features 
Other _ 


.  Friendly  icons  (individual  platforms) 
,  Enemy  icons 
,  Overlays 


Please  continue  to  the  next  page 


Section  2 

•  For  each  of  the  following  statements,  circle  whether  you  STRONGLY  AGREE,  AGREE,  DISAGREE,  or 
STRONGLY  DISAGREE  with  the  statements,  or  circle  DON’T  KNOW  if  you’re  not  sure. 

6.  ‘Tm  never  sure  what  FBCB2  will  do  next" 


Don’t  Strongly  Disagree  Agree 

Know  Disagree 

7.  “FBCB2  really  helps  me  do  my  job" 

Don’t  Strongly  Disagree  Agree 

Know  Disagree 


Strongly 

Agree 


Strongly 

Agree 


8.  “I  don’t  use  FBCB2  if  I  can  avoid  it” 

Don’t  Strongly  Disagree  Agree 

Know  Disagree 

9.  “Digitization  isn’t  worth  the  amount  of  effort  I  have  to  put  into  if 

Don’t  Strongly  Disagree  Agree 

Know  Disagree 

10.  “There  are  only  a  few  tasks  FBCB2  is  useful  for” 

Don’t  Strongly  Disagree  Agree 

Know  Disagree 


Strongly 

Agree 


Strongly 

Agree 


Strongly 

Agree 


1 1 .  “The  information  on  FBCB2  is  almost  always  correct,  so  I  don’t  bother  checking  if 

Don’t  Strongly  Disagree  Agree 

Know  Disagree 


Strongly 

Agree 


12.  “Even  when  it’s  not  working  perfectly,  digitization  is  better  than  analog” 


Don’t  Strongly  Disagree 

Know  Disagree 


Agree 


Strongly 

Agree 


Please  continue  to  the  next  page 
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13.  “As  long  as  I  check  the  information  you  get  from  FBCB2  to  make  sure  it’s  correct,  digitization  is  really  usefuF 


Don’t  Strongly  Disagree  Agree  Strongly 

Know  Disagree  Agree 

14.  “Sometimes  I  make  up  my  own  shortcuts  that  get  things  done  faster” 

Don’t  Strongly  Disagree  Agree  Strongly 

Know  Disagree  Agree 


Section  3 

•  For  each  of  the  following  questions,  please  circle  your  answer. _ 

15.  How  often  would  you  look  at  the  FBCB2  display  after  you  crossed  the  Line  of  Departure  (LD)?  (Circle) 

If  I’m  lost/confused  or  when  approaching  the  next  decision  point  Never 

Every  few  minutes  Every  few  seconds  Other _ 

16.  How  often  would  you  hand-off  the  requirement  to  monitor  the  FBCB2  display  to  someone  else?  (circle 
one) 

Never  If  I’m  doing  something  else  Every  few  minutes 

Always  Other _ 


Section  4 

•  Please  answer  the  questions  by  filling  in  the  blank. _ 

17.  If  you  wanted  to  hand-off  monitoring  the  FBCB2  display  to  someone  else,  who  would  you  hand  it  off  to? 


18.  What  feature  of  FBCB2  is  the  most  difficult  to  work  with? 


You’re  done!  Return  this  questionnaire  to  the  envelope. 

The  purpose  of  this  research  is  to  find  out  how  people  learn  to  use  digitization.  This  research  will 
help  us  to  find  the  best  ways  to  train  and  evaluate  the  use  of  digital  systems.  Your  participation  has  made 
this  research  possible  THANKS! 
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